Automatically Evaluating Answers to Definition Questions
نویسندگان
چکیده
Following recent developments in the automatic evaluation of machine translation and document summarization, we present a similar approach, implemented in a measure called POURPRE, for automatically evaluating answers to definition questions. Until now, the only way to assess the correctness of answers to such questions involves manual determination of whether an information nugget appears in a system’s response. The lack of automatic methods for scoring system output is an impediment to progress in the field, which we address with this work. Experiments with the TREC 2003 and TREC 2004 QA tracks indicate that rankings produced by our metric correlate highly with official rankings, and that POURPRE outperforms direct application of existing metrics.
منابع مشابه
LAMP - TR - 119 CS - TR - 4695 UMIACS - TR - 2005 - 04 February 2005 Automatically Evaluating Answers to Definition Questions
Following recent developments in the automatic evaluation of machine translation and document summarization, we present a similar approach, implemented in a measure called Pourpre, for automatically evaluating answers to definition questions. Until now, the only way to assess the correctness of answers to such questions involves manual determination of whether an information nugget appears in a...
متن کاملEvaluating EFL Learners’ Philosophical Mentality through their Answers to Philosophical Questions: Using Smith’s Framework
Given the role philosophical mentality can fulfill in bringing individuals the essential skills of wisdom and well thinking, the present paper, by applying Smith’s (2007) theoretical framework, strived to explore the extent philosophic-mindedness exists among the participants. Considering the fact that, a philosophic mind begets philosophical answers, the participants’ philosophical thi...
متن کاملA Practically Unsupervised Learning Method to Identify Single-Snippet Answers to Definition Questions on the Web
We present a practically unsupervised learning method to produce single-snippet answers to definition questions in question answering systems that supplement Web search engines. The method exploits on-line encyclopedias and dictionaries to generate automatically an arbitrarily large number of positive and negative definition examples, which are then used to train an SVM to separate the two clas...
متن کاملLearning to Identify Single-Snippet Answers to Definition Questions
We present a learning-based method to identify single-snippet answers to definition questions in question answering systems for document collections. Our method combines and extends two previous techniques that were based mostly on manually crafted lexical patterns and WordNet hypernyms. We train a Support Vector Machine (SVM) on vectors comprising the verdicts or attributes of the previous tec...
متن کاملQualitative Dimensions in Question Answering: Extending the Definitional QA Task
Current question answering tasks handle definitional questions by seeking answers which are factual in nature. While factual answers are a very important component in defining entities, a wealth of qualitative data is often ignored. In this incipient work, we define qualitative dimensions (credibility, sentiment, contradictions etc.) for evaluating answers to definitional questions and we explo...
متن کامل